Course Project Notebook: Analysis of Crunchbase¶

Executive Summary¶

Our main goal is to study trends and patterns of big companies as well as upcoming startups in tech. The startup culture has lead to an enormous growth in the tech industry and generated some of the most innovative products and services in history.

We have collected data from a website called Crunchbase which lists all businesses and startups in the tech industry and provides information on their respective category, investments, fundings, acquisitions and IPOs among many other features.

We aim to cluster VC and investor groups in certain categories, predict funding potential for upcoming startups and investment opportunities for investors. We will also use various models for feature selection to predict the aforementioned characteristics. Overall, our objective is to gain a startup as well as VC-side perspective of funding from crunchbase data.

We plan to use k-means and LSI for clustering investors and try a variety of regression approaches such as linear regression, GLMs or GAMs to determine funding and investment potential.

Team Members¶

Ansuya Ahluwalia (ansuya@cs.ucla.edu) -- exploratory data analysis, graph mining

Ashwini Bhatkhande (ash@cs.ucla.edu) -- machine learning, production of final notebook

Raghav Mehrish (rmehrish@ucla.edu) -- data extraction, machine learning

Shivin Kapur (shivinkapur@ucla.edu) -- data modeling, production of final notebook

Data¶

Companies
Acquisitions
Rounds
Investments

Tools and Packages¶

Exploratory Data Analysis

Python : Pandas, numpy, matplotlib.pyplot
R : data.table, plyr, ggplot2, reshape2

Data Modeling

R : rmisc, MASS, plyr, splines, data.table

Predictive Modeling

Python : pandas, numpy, sklearn, pylab
R : caret, kernlab, plyr, e1071, MASS, nnet, knn3

Graph Mining

R : igraph, Matrix

Exploratory Data Analysis¶

%load_ext rmagic
import rpy2 as Rpy

#Histogram of company categories

/Users/Work/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py:1070: DtypeWarning: Columns (9) have mixed types. Specify dtype option on import or set low_memory=False.
  data = self._reader.read(nrows)

<matplotlib.axes.AxesSubplot at 0x108d38d50>

#Total Funding per Company Category

<matplotlib.legend.Legend at 0x107b619d0>

#Frequency of Status

<matplotlib.axes.AxesSubplot at 0x10672cd50>

#Histogram of founded year

<matplotlib.axes.AxesSubplot at 0x1066b0550>

#Total funding with funding quarter and no. of companies

      company_category_code company_country_code funding_round_type funded_year
11635                   web                  USA            venture        2010
11636                   web                  USA            venture        2011
11637                   web                  USA            venture        2012
11638                   web                  USA            venture        2013
11639                   web                  USA            venture        2014
11640                   web                  ZAF           series-a        2012
      total_amount
11635    244010490
11636    416615077
11637    191888230
11638    121948356
11639    170653603
11640       600000

# Problems faced during feature extraction

# Merged table with important data

 [1] "name"                   "permalink"              "homepage_url"          
 [4] "category_code"          "funding_total_usd"      "status"                
 [7] "country_code"           "state_code"             "region"                
[10] "city"                   "funding_rounds"         "founded_at"            
[13] "founded_year"           "investor_permalink"     "investor_name"         
[16] "investor_category_code" "investor_country_code"  "investor_state_code"   
[19] "investor_region"        "investor_city"          "funding_round_type.y"  
[22] "funded_at.y"            "funded_month.y"         "funded_quarter.y"      
[25] "funded_year.y"          "raised_amount_usd.y"    "quarters"

# All important features

 [1] "category_code"          "status"                 "country_code"          
 [4] "funding_rounds"         "founded_year"           "investor_category_code"
 [7] "investor_country_code"  "funding_round_type.y"   "funded_year.y"         
[10] "quarters"               "labelNum"

Predictive Modeling¶

%%R

trainClass = read.csv('/Users/raghav297/dropbox/Documents/UCLA/UCLA_Spring_14/CS249/Crunchbase_Data/trainClass.csv')
testDescr = read.csv('/Users/raghav297/dropbox/Documents/UCLA/UCLA_Spring_14/CS249/Crunchbase_Data/testDescr.csv')

trainDescr = read.csv('/Users/raghav297/dropbox/Documents/UCLA/UCLA_Spring_14/CS249/Crunchbase_Data/trainDescr.csv')
testClass = read.csv('/Users/raghav297/dropbox/Documents/UCLA/UCLA_Spring_14/CS249/Crunchbase_Data/testClass.csv')

trainDescr$X = NULL
trainClass$X = NULL
testDescr$X = NULL
testClass$X = NULL

# Column names of training and testing descriptors and targets
# Dimensions of training and testing descriptors and targets

 [1] "category_codeadvertising"      "category_codebiotech"         
 [3] "category_codeecommerce"        "category_codeenterprise"      
 [5] "category_codemobile"           "category_codesoftware"        
 [7] "category_codeweb"              "statusoperating"              
 [9] "country_codeUSA"               "funding_rounds"               
[11] "founded_year"                  "investor_country_codeUSA"     
[13] "funding_round_type.yseries.a"  "funding_round_type.yseries.b" 
[15] "funding_round_type.yseries.c." "funding_round_type.yventure"  
[17] "funded_year.y"                 "quartersQ2"                   
[19] "quartersQ3"                    "quartersQ4"                   
 [1] "category_codeadvertising"      "category_codebiotech"         
 [3] "category_codeecommerce"        "category_codeenterprise"      
 [5] "category_codemobile"           "category_codesoftware"        
 [7] "category_codeweb"              "statusoperating"              
 [9] "country_codeUSA"               "funding_rounds"               
[11] "founded_year"                  "investor_country_codeUSA"     
[13] "funding_round_type.yseries.a"  "funding_round_type.yseries.b" 
[15] "funding_round_type.yseries.c." "funding_round_type.yventure"  
[17] "funded_year.y"                 "quartersQ2"                   
[19] "quartersQ3"                    "quartersQ4"                   
[1] "x"
[1] "x"
[1] 164606     20
[1] 54865    20
[1] 164606      1
[1] 54865     1

# Sample output for one of the classifiers

['category_codeadvertising' 'category_codebiotech' 'category_codeecommerce'
 'category_codeenterprise' 'category_codemobile' 'category_codesoftware'
 'category_codeweb' 'statusoperating' 'country_codeUSA' 'funding_rounds'
 'founded_year' 'investor_country_codeUSA' 'funding_round_type.yseries-a'
 'funding_round_type.yseries-b' 'funding_round_type.yseries-c+'
 'funding_round_type.yventure' 'funded_year.y' 'quartersQ2' 'quartersQ3'
 'quartersQ4']
['category_codeadvertising' 'category_codebiotech' 'category_codeecommerce'
 'category_codeenterprise' 'category_codemobile' 'category_codesoftware'
 'category_codeweb' 'statusoperating' 'country_codeUSA' 'funding_rounds'
 'founded_year' 'investor_country_codeUSA' 'funding_round_type.yseries-a'
 'funding_round_type.yseries-b' 'funding_round_type.yseries-c+'
 'funding_round_type.yventure' 'funded_year.y' 'quartersQ2' 'quartersQ3'
 'quartersQ4']
Random Forest
Pred  [1 5 5 ..., 2 2 1]
Mean : 0.909742
Feature Importances  [ 0.01507738  0.01305174  0.01320477  0.01932507  0.01867471  0.02129487
  0.0117261   0.03326322  0.02709368  0.19284599  0.22825277  0.02048339
  0.03373769  0.03094636  0.04350878  0.0150039   0.16409708  0.03255682
  0.03360408  0.03225159]
Predict Probability  [[ 0.82431818  0.17568182  0.         ...,  0.          0.          0.        ]
 [ 0.6         0.275       0.025      ...,  0.          0.          0.        ]
 [ 0.59044289  0.33256022  0.07699689 ...,  0.          0.          0.        ]
 ..., 
 [ 0.          0.          0.         ...,  0.          0.          1.        ]
 [ 0.          0.          0.         ...,  0.          0.          1.        ]
 [ 0.          0.          0.         ...,  0.          0.          1.        ]]
Transform  [[-1.37108833  1.28185539  0.72992665]
 [-1.37108833  0.03388946 -1.4315749 ]
 [-1.37108833  1.28185539  1.0901769 ]
 ..., 
 [ 1.72109748  0.24188378  0.00942613]
 [ 1.72109748  0.24188378  0.00942613]
 [ 1.72109748  0.24188378  0.36967639]]

Classification Output¶

knn¶

Trees

SVM

Logistic Regression

Naive Bayes

Ensemble Methods¶

Random Forest

Adaboost

Extra Trees¶

Gradient Boosting¶

Comparison between classifiers¶

	Accuracy
knn	0.837419
Trees	0.910234
SVM	0.542404
Logistic Regression	0.425043
Naive Bayes	0.304602
Random Forest	0.909469
Adaboost	0.910289
Extra Trees	0.909833
Gradient Boosting	0.475987

# Accuracies of various classifiers compared to each other

Analyzing knn, Trees, Random Forest¶

knn¶

Trees¶

Random Forest¶

Graph Mining¶

Adjacency, Two-mode Incidence, Similarity and Overlap Network Graphs for investors and the companies invested in.¶

from IPython.display import Image

path='/Users/Work/Documents/CS249/project/crunchbase/plots/final/png/'
files = ['Acompanies_top110companies.png',
         'Ainvestors_top10companies.png',
         'iIC_top110companies.4.png',
         'Heatmap_Top20Companies.png',
        'Jaccard_Similarity_Companies.png',
        'Jaccard_Similarity_Investors.png',
        'olInvestor1gcustomlayoutRand5.png'
         ]

for file in files:
    #print('{0}{1}'.format(path, file))
    display(Image(filename='{0}{1}'.format(path, file)))

Adjacency and Two-mode Incidence Graphs for acquirers and the companies acquired.¶

path='/Users/Work/Documents/CS249/project/crunchbase/plots/final/png/'
files = ['Aacqcompanies_top1000companies.png',
         'Aacquirer_top1000companies.png',
         'iAC_top1000companies.png']

for file in files:
    display(Image(filename='{0}{1}'.format(path, file)))

Adjacency, Two-mode Incidence and Overlap Network Graphs for acquirer regions and regions of companies acquired.¶

path='/Users/Work/Documents/CS249/project/crunchbase/plots/final/png/'
files = ['AacqcompanyRegions_top1000companies.png',
        'AacquirerRegions_top1000companies.png',
        'iACRegions_top1000.png',
        'olAcqRegions1gcustomlayoutTop1000.png']

for file in files:
    display(Image(filename='{0}{1}'.format(path, file)))